World models are AI tools that understand the dynamics of the real world, including physics and spatial properties. They can use input data, including text, image, video, sound, and movement, to predict what happens next.

World models have gone from hand-coded rules to billion-parameter neural networks, but the underlying goal has stayed the same: Teach AI how to behave in the real world. 

Traditional approaches relied on engineers explicitly programming physical rules. They were precise in narrow conditions and useless outside them. They worked well inside controlled environments like game engines and early robotics simulators, but fell apart the moment the real world threw something unexpected at them. Modern world models learn those rules from data.

With generative AI, approaches transformed entirely. Instead of hard coding rules, developers trained models with internet-scale datasets. When prompted, these models can generate synthetic high-fidelity worlds. 

A new generation of world foundation models are now pretrained on massive real-world and infinite synthetic data, not just to generate but also reason and predict based on physics laws. A pretrained foundation model handles the heavy lifting; targeted post-training on proprietary data handles the rest—cutting development from years to months.

Building a world foundation model (WFM) involves these steps:

Data curation is a crucial step for pretraining and continuous training of world models, especially when working with large-scale multimodal data. It involves processing steps like filtering, annotation, classification, and deduplication of image or video data to ensure high quality when training or post-training highly accurate models.

In video processing, data curation starts with splitting and transcoding the video into smaller segments, followed by quality filtering to retain the high-quality data. State-of-the-art vision language models are used to annotate key objects or actions, while video embeddings help semantic deduplication to remove redundant data.

The data is then organized and cleaned for training. Throughout this process, efficient data orchestration ensures a smooth data flow among the GPUs, enabling them to handle large-scale data and achieve high throughput.

Tokenization converts high-dimensional visual data into smaller units called tokens, facilitating machine learning processing. Tokenizers transform pixel redundancies in images and video into compact, semantic tokens, enabling efficient training of large-scale generative models and inference on limited resources. There are two main methods:

• Discrete Tokenization: Represents images and videos as integers.
• Continuous Tokenization: Represents images and videos as continuous vectors.

This approach enhances model learning speed and performance.

Building foundation models starts with choosing an architecture design and training with massive data for a task objective. The transformer is the backbone of modern world models, but there are two distinct ways to use it, and each has different strengths:

• Autoregressive transformers generate the world token by token, each frame conditioned on everything that came before. Like a language model predicting the next word, an autoregressive world model predicts the next visual state. It is naturally suited to sequential decision-making and long-horizon planning. It understands cause and effect in time.
• Diffusion transformers start with noise and progressively denoise it into a coherent, photorealistic world. Rather than generating sequentially, they refine the entire output simultaneously and produce higher visual fidelity and better spatial consistency. They excel at generating rich synthetic environments.

Each approach decomposes a complex world generation problem into smaller, tractable steps.

Real-world data is costly to capture and hard to scale. Physical agents like robots, autonomous vehicles, smart cities, and industries need to learn to operate across environments, tasks, and conditions. Synthetic data generation is critical to match the scale. Continuous training on synthetic data helps world models evolve and predict what happens next, even for the situations they have never seen before. 

Enables Closed-Loop Learning

Robots can train, fail, and improve inside a world model without physical risk or cost by running thousands of reinforcement learning iterations in simulation that would be impossible to execute on real hardware.

Generalizes Across Embodiments and Domains

A single foundation model can be post-trained for a policy for humanoids, autonomous vehicles, surgical robots, and industrial arms rather than training a separate model from scratch for every new embodiment or environment.

Transfers Simulation to Reality

The sim-to-real gap has historically broken policies trained in simulation. World models convert physics-based simulation outputs into photorealistic environments, closing that gap so policies trained synthetically hold up in deployment.

Accelerates AI Model Training

Starting from a pretrained world foundation model means developers don’t build from scratch and they inherit physics understanding, spatial reasoning, and temporal coherence, then post-train on domain-specific data to reach production performance faster.

World models, when used with 3D simulators, serve as virtual environments to safely streamline and scale training for autonomous machines. With the ability to generate, curate, and encode video data, developers can better train autonomous machines to sense, perceive, and interact with dynamic surroundings.

World models bring significant benefits to every stage of the autonomous vehicle (AV) pipeline. With pre-labeled, encoded video data, developers can curate and train the AV stack to recognize the behavior of vehicles, pedestrians, and objects more accurately. These models can create predictive video simulations based on text and visual inputs and generate new scenarios, such as different traffic patterns, road conditions, weather, and lighting, to post-train the reasoning vision-language-action model powering the vehicle and accelerate testing and validation.

World models generate photorealistic synthetic data and predictive world states to help robots develop spatial intelligence. Using virtual simulations powered by physical simulators, these models let robots practice tasks safely and efficiently, accelerating learning through rapid testing and training. They help robots adapt to new situations by learning from diverse data and experiences.

Modified world models enhance planning by simulating object interactions, predicting human behavior, and guiding robots to reach goals accurately. They also enhance decision-making by conducting multiple simulations and learning from the feedback. With virtual simulations, developers can reduce real-world testing risks, cutting time, costs, and resources.

Trained with rich, multimodal data and advanced reasoning capabilities, world models can perform complex video analytics on massive amounts of recorded and live videos. These models enable natural language Q&A, automated summarization, object detection, event localization, and richer contextual understanding of visual content in videos—capabilities that surpass traditional computer vision methods. World models also generate photorealistic synthetic data on corner cases, helping to better train AI models to detect critical incidents.

Common applications of world models for video analytics are found in both industrial and smart city settings to improve safety and operational efficiency. Examples include identifying injury risks and unsafe behaviors for industrial safety, providing a detailed cause-and-effect understanding for rapid incident investigation, monitoring traffic, crowd flows, public safety incidents, and environmental hazards in smart cities, and identifying defects and irregularities on manufacturing lines through visual inspection for quality control.